Two things happened in 2017:

Linux finally got native, working MPLS (L3VPN) and VRF support. 3 years later, a thorough documentation of MPLS configuration on Linux is still largely missing. Recently, after digging into all kinds of codes and documentation, I had a standard MPLS core network up and running in my lab. This article is a write-up for my lab setup.

Those IP routing guys without the experience of large-scale carrier network may be unfamiliar with the MPLS technology. MPLS (L3VPN) is like VLAN trunking but for layer 3: you put an interface of a router into a VRF instance (like a VLAN); every router can have multiple routing tables, one for every VRF (like ARP tables for VLANs); when core routers routing packets, these packets will be prefixed by MPLS labels (like VLAN tags) so different packets from different routing tables do not interfere. MPLS brings a bunch of good things:

All good protocols come with some requirements. MPLS is a “layer 2.5” protocol, which means it runs on a layer 2 link. For MPLS to work, all your core routers need to be connected via either a layer 2 link (e.g. Ethernet), or a non-layer-2 tunnel that explicitly supports MPLS (via RFC2547bis, for example, GRE or L2TP). MPLS typically have a header of 8 octets, so you might want to set a slightly larger MTU.

All routers in a MPLS core network are loosely divided into 2 classes: P (Provider) routers are those who doesn’t connect to a customer device; PE (Provider Edge) routers are those have at least one connection to a customer device. A customer router connecting to a PE is called a CE (Customer Edge) router. I’m not going to mention more theories here but to focus on get simple things running, so these are the steps to start an MPLS core network:

Here I’m going to demonstrate how to configure all the steps under Linux using a minimal lab environment.

Here’s a classic topology to demonstrate MPLS’s capability to provide transit to customers with overlapping IP addresses. Say we are a regional small ISP and provide a service to connect customer sites located in 2 cities. One day 2 customers (the “customer1” and the “customer2”) purchased our transit service but their LAN IP address ranges are the same. If we use the traditional IP core network, we will be at least losing one customer. But with an MPLS core network, we can easily satisfy both customers.

So here is our topology. Notes:

To limit the length of this article, we are not discussing the following advanced topics:

My friend LittleWolf wrote an excellent article covering some topics I’ve left out: how to use IS-IS as the iBGP protocol in core, and how to configure BGP between CE and PE routers, all in a similar topology. Read it here: [ Linux ] 使用 Debian Linux 构架 MPLS L3 网络

All devices we use run the following software:

The first line of each code block describes which environment the following code should be typed in:

Install FRRouting:

Launch FRRouting via systemd:

Load MPLS kernel module:

You should be able to ping a router’s interface IP from its direct neighbor router.

The object here is to make all loopback IP addresses reachable from any point of your core network. Because I don’t know how to configure other protocols To simplify things, we use OSPF as an example and put all the routers in Area 0. Note that we configured all the Ethernet connections between routers as point-to-point links, so we need to manually set interface type here.

Any router’s loopback IP address should be pingable from any other router in the core network. For example we ping RR from PE1:

Every core-facing interface should enable MPLS processing and enable LDP protocol:

RR doesn’t really need to be practicing this MPLS network here; it only needs to be able to reach all the PEs. But current LDP daemon implementation of FRRouting does not support Downstream Unsolicited mode so we are left with the only option of Downstream-on-Demand mode. In this mode, if you enable IGP on the RR but do not enable MPLS on it, packets going from PE to RR will be discarded at P because P doesn’t have an entry for RR in its MPLS switching table. Again, to make things simple, we enable MPLS on the RR too:

If you really doesn’t want to run MPLS on the RR, there is another option: do not enable IGP on the RR, write a static route to RR on its neighbor P or PE router and redistribute that static route. The third option is to use label local allocate for to force LDP allocate a label for the prefix to RR.

All adjacent routers should have LDP session established. For example, on the P:

You should see 3 LDP sessions:

Plus all core routers should still be able to ping each other’s loopback IP. (These pings are now going through MPLS network rather than IP network. That’s a sign that your MPLS network is working.)

iBGP only needs to be configured on PEs but not Ps. This is one of MPLS’s benefits: core routers only need to process label switching but do not need to store and update routing tables. You can use either a full mesh iBGP setup or a Route Reflector; we have a RR on the topology, so we are sticking to the RR method. To simplify things, I’m only demonstrating IPv4 config (disable IPv4 unicast SAFI, enable IPv4 VPNv4 SAFI); if you need IPv6 inside MPLS then repeat the config for IPv6.

In a production environment, a single RR instance is a single point of failure. Packet Pusher authored a good article about RR redundancy architecture designing for reference.

All BGP sessions should come up.

There are 2 ways to implement VRF on Linux, net namespace and VRF interface. Net namespace is a network stack abstraction unique to the Linux kernel, it provides complete isolation which is useful for paravirtualization and containers, but it also prevents route leaking and data exchange so we are not going to use it. VRF interface is more like the VRF you would see on a commercial router system, since we are only using Linux as a router, we choose this method.

The config is quite simple. Every VRF interface will be associated to a routing table identified by its table ID during creation. Then you can add routed interfaces to a VRF just like how you add a switched port to a bridge. To make the new routing table available to an application, here we will add an unreachable default route with its metric set to the max value.

To prevent human errors, we use the same VRF name and table ID for the same customer on every PE router.

This article is way too long as what I expected, so we are going to redistribute connected route only just to keep things simple. Dynamic routing protocols between PE and CE routers can be configured in the same way. In this step, you need to create an empty BGP configuration on every VRF, import all the routes you need to redistribute to that BGP instance’s RIB, add a route distinguisher community to the route, then import it to the main BGP instance’s RIB.

To prevent human error, we use the same RT and RD as the table ID for every customer, but you need to be aware that they can all be different if required. Another thing you need to remind yourself is that although FRRouting vtysh tried hard to mimic a grammar like the Cisco IOS, they still differ in some way.

When you are typing the config, FRRouting will alert you about some error; don’t panic and continue.

Now we should see connected routes redistributed by BGP in every VRF of every PE. For example, on PE1:

You should see something like this:

The first thing you need to check is the routers of the same customer pings fine. As we have PHP and TTL propagation enabled, 3 MPLS hops in the core will result in 2 hops of non-IP routing. So, if you use traceroute or mtr, you’ll see something like this:

Sadly, Linux only support RFC4884 partially and doesn’t support RFC4950 at the time of writing, so traceroute -e and mtr --mpls is not going to help you at all.

Next, we need to verify that we do correctly connected the sites of the same customer. To do this we use netcat to simulate a simple TCP chat service. Start the server on one of the customer sites:

And connect to the server on another site of the same customer:

After a successful connection, you can type anything into both the server and the client, press enter, and the message should be sent to the other side.

The whole story of Linux development can be summed up in one story: someday you want to make a model plane, so you built one using sticks and paper in the backyard. The next day when you woke up, you are surprised to find that the model plane is already in the sky flown by your neighbor carrying tens of passengers, and there are two people standing on the wings trying to fix the engine and maintain balance as well.

“It is provided free of charge, and if it fails, I won’t be hurt, so why to bother?” And you went on your daily work.

Thanks:

Two CCIE certified friends helped a lot during the writing of this article.

References: